ISA 444/544: Business Forecasting

.title[
# ISA 444/544: Business Forecasting
]
.subtitle[
## 11: Forecasting Environment and Baseline Forecasts
]
.author[
### <br>Fadel M. Megahed, PhD <br><br> Raymond E. Glos Professor in Business <br> Farmer School of Business<br> Miami University<br><br> <a href="https://twitter.com/FadelMegahed"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z" /></svg> <span class="citation">@FadelMegahed</span></a> <br> <a href="https://github.com/fmegahed/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z" /></svg> fmegahed</a> <br> <a href="mailto:fmegahed@miamioh.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z" /></svg> fmegahed@miamioh.edu</a><br> <a href="https://calendly.com/fmegahed"><svg viewBox="0 0 384 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M202.021 0C122.202 0 70.503 32.703 29.914 91.026c-7.363 10.58-5.093 25.086 5.178 32.874l43.138 32.709c10.373 7.865 25.132 6.026 33.253-4.148 25.049-31.381 43.63-49.449 82.757-49.449 30.764 0 68.816 19.799 68.816 49.631 0 22.552-18.617 34.134-48.993 51.164-35.423 19.86-82.299 44.576-82.299 106.405V320c0 13.255 10.745 24 24 24h72.471c13.255 0 24-10.745 24-24v-5.773c0-42.86 125.268-44.645 125.268-160.627C377.504 66.256 286.902 0 202.021 0zM192 373.459c-38.196 0-69.271 31.075-69.271 69.271 0 38.195 31.075 69.27 69.271 69.27s69.271-31.075 69.271-69.271-31.075-69.27-69.271-69.27z" /></svg> Automated Scheduler for Office Hours</a><br><br>
]
.date[
### Fall 2025
]

---

## Learning Objectives for Today's Class

- Install and import Nixtla's libraries ([StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html), [MLForecast](https://nixtlaverse.nixtla.io/mlforecast/index.html), [NeuralForecast](https://nixtlaverse.nixtla.io/neuralforecast/docs/getting-started/introduction.html), [UtilsForecast](https://nixtlaverse.nixtla.io/utilsforecast/index.html), and [TimeGPT](https://nixtlaverse.nixtla.io/nixtla/docs/getting-started/introduction.html)) for forecasting

- Distinguish fixed window from rolling-origin

- Introduce forecast accuracy metrics (MAE, MAPE, RMSE)

- Recognize the importance of baseline forecasts in time series forecasting

- Identify common baseline forecasts (Naive, Seasonal Naive, etc.)

- Implement [baseline models with StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html#baseline-models)

---
class: inverse, center, middle

# The Nixtlaverse Open-Source Forecasting Libraries

---

## Nixtla's Forecasting Libraries

---

## Nixtla's Forecasting Libraries

Nixtla provides **several open-source Python libraries** (and a closed source **TimeGPT** tool accessible via API calls) for **scalable forecasting tasks**. These libraries are **relatively** easy to use and can be integrated into your future forecasting workflows:

- **[StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html)** – Fast & scalable statistical models (`ARIMA`, `ETS`, etc.).
- **[MLForecast](https://nixtlaverse.nixtla.io/mlforecast/index.html)** – Machine learning-based forecasting (e.g., `XGBoost`, `LightGBM`).
- **[NeuralForecast](https://nixtlaverse.nixtla.io/neuralforecast/docs/getting-started/introduction.html)** – Deep learning models for time series (e.g., `NBEATS`, `NHITS`, and `TFT`).
- **[UtilsForecast](https://nixtlaverse.nixtla.io/utilsforecast/index.html)** – Utility functions for plotting, evaluation, etc.
- **[TimeGPT](https://nixtlaverse.nixtla.io/nixtla/docs/getting-started/introduction.html)** – an AI transformer-powered forecasting API that requires minimal tuning.

These libraries enable **forecasting at scale, which you will need in practice**.

**Note:** These libraries can be installed via `pip` and are described in detail in [nixtlaverse.nixtla.io](https://nixtlaverse.nixtla.io/). Use the left-hand navigation bar to explore the documentation for each library.
]

---
class: inverse, center, middle

# Fixed Window vs. Rolling-Origin

---

## The Fixed Window Evaluation Approach

- **Fixed Window** is the simplest approach to splitting your time series data into a training and a testing/holdout set.

- The **goal** is to **train your model on the training set** and **evaluate its performance on the testing set (the last `k` observations in the data)**.  
  + .black[.bold[Note that this is quite different than traditional machine learning applications for cross sectional data.]]

- The evaluation on the **testing/holdout** set can serve two purposes:  
  + **Model Evaluation**: Assess the model's **performance on unseen data** (since it is not used during training, and hence acts as a proxy for the model's performance on future data).  
  + **Model Selection**: Compare the **performance of different models** to select the **best one**.   
     - Note that using this approach for model selection is **reasonable if your models do not involve hyperparameter tuning** (otherwise, you may overfit to the testing set).

---

## The Fixed Window Evaluation Approach

---

## The Rolling-Origin Evaluation Approach

- **Rolling-Origin Evaluation** is a method for splitting time series data into training and testing sets where the testing sets move forward over time.

- The **goal** is to **train the model on a subset of past observations** and evaluate its **performance on a future testing set at multiple time steps**.

- The key difference from a fixed window approach is that the **testing set shifts forward, allowing for multiple evaluations**:  
  + The **training set may expand (expanding window)** or **remain fixed (rolling window)**.  
  + This ensures that model performance is assessed across **different points in time**.  
  + It **reduces sensitivity to the initial split point** and provides a more **robust evaluation** of model performance over time.

---

## Expanding Window Evaluation (By 1 Month)

---

## Expanding Window Evaluation (By 12 Month)

---

## Rolling Non-Expanding Window Evaluation

---

## Cross Validation within the Nixtlaverse

**Note:** The `cross_validation` method can also be applied to other NixtlaForecast objects (e.g., `MLForecast`, `NeuralForecast`) to perform cross-validation for machine learning and deep learning models. See the [StatsForecast Cross Validation Tutorial](https://nixtlaverse.nixtla.io/statsforecast/docs/tutorials/crossvalidation.html) to access the page shown above.
]

---

## Activity: The `cross_validation` Method

---

## Recap of Fixed vs. Rolling-Origin

- **Fixed Window**:  
  + **Simplest approach** to splitting data into training and testing sets.  
  + **Testing set is fixed** and **does not move forward** over time.  
  + **Provides a single evaluation** of model performance.
  
- **Rolling-Origin**:
  + **Testing set moves forward** over time.  
  + **Training set may expand or remain fixed**.  
  + **Provides multiple evaluations** of model performance.  
  + **Reduces sensitivity to the initial split point**.  
  + **More robust evaluation** of model performance over time.

---

## Recap of Fixed vs. Rolling-Origin

- In practice, the **rolling-origin approach is preferred** for time series forecasting tasks since it mimics the real-world scenario of forecasting future data points. The choice of:  
  + **Expanding vs. Rolling Window**,
  + **Window Size**, and
  + **Step Size**
depends on the specific forecasting task and the data at hand.

---

# Forecast Accuracy Metrics

---

## Model Performance Evaluation in the Nixtlaverse

``` python
from utilsforecast.losses import *
```

### `evaluate`

``` python
evaluate (df:~AnyDFType, metrics:List[Callable],
           models:Optional[List[str]]=None,
           train_df:Optional[~AnyDFType]=None,
           level:Optional[List[int]]=None, id_col:str='unique_id',
           time_col:str='ds', target_col:str='y',
           agg_fn:Optional[str]=None)
```

**Source:** [Nixtla's UtilsForecast Evaluation Documentation](https://nixtlaverse.nixtla.io/utilsforecast/evaluation.html)
]

---

## Model Performance Evaluation in the Nixtlaverse (Cont.)

.font80[
<table><thead><tr><th></th><th><strong>Type</strong></th><th><strong>Default</strong></th><th><strong>Details</strong></th></tr></thead><tbody><tr><td>df</td><td>AnyDFType</td><td></td><td>Forecasts to evaluate.<br/>Must have <code>id_col</code>, <code>time_col</code>, <code>target_col</code> and models’ predictions.</td></tr><tr><td>metrics</td><td>List</td><td></td><td>Functions with arguments <code>df</code>, <code>models</code>, <code>id_col</code>, <code>target_col</code> and optionally <code>train_df</code>.</td></tr><tr><td>models</td><td>Optional</td><td>None</td><td>Names of the models to evaluate.<br/>If <code>None</code> will use every column in the dataframe after removing id, time and target.</td></tr><tr><td>train_df</td><td>Optional</td><td>None</td><td>Training set. Used to evaluate metrics such as <a href="https://Nixtla.github.io/utilsforecast/losses.html#mase" target="_blank" rel="noreferrer"><code>mase</code></a>.</td></tr><tr><td>level</td><td>Optional</td><td>None</td><td>Prediction interval levels. Used to compute losses that rely on quantiles.</td></tr><tr><td>id_col</td><td>str</td><td>unique_id</td><td>Column that identifies each serie.</td></tr><tr><td>time_col</td><td>str</td><td>ds</td><td>Column that identifies each timestep, its values can be timestamps or integers.</td></tr><tr><td>target_col</td><td>str</td><td>y</td><td>Column that contains the target.</td></tr><tr><td>agg_fn</td><td>Optional</td><td>None</td><td>Statistic to compute on the scores by id to reduce them to a single number.</td></tr><tr><td><strong>Returns</strong></td><td><strong>AnyDFType</strong></td><td></td><td><strong>Metrics with one row per (id, metric) combination and one column per model.<br/>If <code>agg_fn</code> is not <code>None</code>, there is only one row per metric.</strong></td></tr></tbody></table>
]

**Source:** [Nixtla's UtilsForecast Evaluation Documentation](https://nixtlaverse.nixtla.io/utilsforecast/evaluation.html)
]

---

## Losses

The most important train signal is the forecast error, which is the difference between the observed value `$y_{\tau}$` and the prediction `$\hat{y}_{\tau}$`, at time `$y_{\tau}$`:

`$$e_{\tau} = y_{\tau} - \hat{y}_{\tau} \quad \quad \tau \in \{t+1, \dots, t+H\}$$`

The train loss summarizes the forecast errors in different evaluation metrics.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Scale-Dependent Errors: `mae`

**MAE** measures the relative prediction accuracy by averaging the absolute deviations between forecasts and actual values.

`$$\text{MAE}(y_{\tau}, \hat{y}_{\tau}) = \frac{1}{H} \sum_{\tau = t+1}^{t+H} |y_{\tau} - \hat{y}_{\tau}|$$`
- **Interpretation**: Provides a straightforward measure of forecast accuracy; lower MAE indicates better performance.

- **Characteristic:** Does not penalize larger errors more than smaller ones; treats all errors equally.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Scale-Dependent Errors: `rmse`

**RMSE** is the square root of the average of the squared differences between forecasts and actual values.

`$$\text{RMSE}(y_{\tau}, \hat{y}_{\tau}) = \sqrt{\frac{1}{H} \sum_{\tau = t+1}^{t+H} (y_{\tau} - \hat{y}_{\tau})^2}$$`

- **Interpretation**: Emphasizes larger errors due to squaring; useful when large errors are particularly undesirable.

- **Characteristic:**  Penalizes large errors more than MAE (i.e., more sensitive to ourliers compared to MAE).

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---

## Percentage Errors: `mape`

**MAPE** calculates the average absolute error as a percentage of actual values.

`$$\text{MAPE}(y_{\tau}, \hat{y}_{\tau}) = \frac{1}{H} \sum_{\tau = t+1}^{t+H} \left| \frac{y_{\tau} - \hat{y}_{\tau}}{y_{\tau}} \right|$$`
**Interpretation**: Expresses forecast accuracy as a percentage; lower MAPE indicates better performance.

**Characteristic:** Can be misleading if actual values are close to zero, leading to extremely high MAPE values.

**Source:** [Nixtla's UtilsForecast Losses Documentation](https://nixtlaverse.nixtla.io/utilsforecast/losses.html)
]

---
class: inverse, center, middle

# Importance of Baseline Forecasts

---

## Why do we need Baseline Forecasts?

- Baseline forecasts serve as **important benchmarks**:

- They **provide a performance "floor"** for more complex models.

- Help **assess if advanced models can add meaningful value**.  
  
- They are **easy to interpret** and **require minimal computational resources**. 
]

]

---
class: inverse, center, middle

# Common Baseline Forecasts

---

## Historic Mean/Average

- **Historic Mean/Average**: The simplest baseline forecast is to use the average of the historical data as the forecast for all future periods.

- **Mathematical Representation**: `$\hat{y}_{t+h} = \frac{1}{t} \sum_{i=1}^{t} y_i$`

- **Assumptions about the Data Generating Process (DGP)**:

- Future values will be similar to the past.

- The data generating process is **relatively stationary (fixed mean with random noise)**, with **no seasonality** and **no trends** (obviously since the mean is fixed).

---

## Historic Mean/Average: Fixed Window

**Notes**: The plot above shows the historic average forecast for the retail trade data. The light gray region represents the training data, while the red region represents the test data. The vertical dashed line indicates the split point between the training and test data. The training data is used to fit the model, while the test data is used to evaluate the model's performance.

**Obviously, the historic average forecast is a straight line, as it is the same for all future periods. Furthermore, it is a terrible choice for this dataset since it clearly has a positive trend.**
]

---

## Naive Forecast

- **Naive Forecast**: The naive forecast assumes that the future value will be the same as the last observed value.

- **Mathematical Representation**: `$\hat{y}_{t+h} = y_t$`

- **Assumptions about the Data Generating Process (DGP)**:

- Future values will be similar to the most recent past value.

- The data generating process can either be **relatively stationary** or **follow a random walk**.

---

## Naive Forecast: Fixed Window

**Notes**: The plot above shows the naive forecast for the retail trade data. The light gray region represents the training data, while the red region represents the test data. The vertical dashed line indicates the split point between the training and test data. The training data is used to fit the model, while the test data is used to evaluate the model's performance.

**The naive forecast lags the training data, and is a straightline for the holdout dataset since we do not update the naive forecast for the fixed window approach.** **Similar to the historic average forecast, the naive forecast is a terrible choice for this dataset since it clearly has a positive trend.**
]

---

## Seasonal Naive Forecast

- **Seasonal Naive Forecast**: The seasonal naive forecast assumes that the future value will be the same as the last observed value from the same season.

- **Mathematical Representation**: `$\hat{y}_{t+h} = y_{t+h-m}$`, where `$m$` is the seasonal period.

- **Assumptions about the Data Generating Process (DGP)**:

- Future values will be similar to the most recent past value from the same season.

- The data generating process will be **seasonal but with no trends**.

---

## Seasonal Naive Forecast: Fixed Window

**Notes**: The plot above shows the seasonal naive forecast for the retail trade data. The light gray region represents the training data, while the red region represents the test data. The vertical dashed line indicates the split point between the training and test data. The training data is used to fit the model, while the test data is used to evaluate the model's performance.

**The seasonal naive forecast lags the training data, and is worse than the naive model (since now the data lags by 12 months instead of 1 month); it would be better if the data is seasonal with no trends.**
]

---
class: inverse, center, middle

# Implementing Baseline Models with StatsForecast

---

## Live Demo

We will now implement the historic average, naive, and seasonal naive forecasts using the [StatsForecast]() library. Our in-class implementation will highlight both the fixed window and rolling-origin approaches.

---

# Recap

---

## Summary of Main Points

By now, you should be able to do the following:

- Distinguish fixed window from rolling-origin

- Introduce forecast accuracy metrics (MAE, MAPE, RMSE)

- Recognize the importance of baseline forecasts in time series forecasting

- Identify common baseline forecasts (Naive, Seasonal Naive, etc.)

- Implement [baseline models with StatsForecast](https://nixtlaverse.nixtla.io/statsforecast/index.html#baseline-models)

---

## 📝 Review and Clarification 📝

1. **Class Notes**: Take some time to revisit your class notes for key insights and concepts.
2. **Zoom Recording**: The recording of today's class will be made available on Canvas approximately 3-4 hours after the session ends.
3. **Questions**: Please don't hesitate to ask for clarification on any topics discussed in class. It's crucial not to let questions accumulate.